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ABSTRACT 

In the context of the development of a possible 
national mathematics assessment, a study was conducted to determine 
whether a test item characterization scheme could be created based on 
a state policy document that serves as the driving force behind 
large-scale performance assessment. Further considerations were 
whether such an item characterization scheme could provide a means to 
differentiate tasks within an assessment and produce equivalent 
assessment forms at various grade levels and whether the item 
characterization scheme could provide information comparable to that 
from previous item characterization approaches. These questions were 
explored using the California Learning Assessment System (CLAS) 
testing program. The Mathematics Framework for California Public 
Schools (CMF) , the state's policy document for mathematics 
assessment, was used to create item characterizations and item 
signatures for CLAS tests. The CMF characterizations were compared to 
three other item characterization methods. No one scheme had a clear 
advantage in itself, but it was advantageous to use the state's own 
policy document as the basis for item characterization, in terms of 
time saved and costs avoided. Appendix A explains the coding schemes, 
and Appendix B presents the specific example of grade-8 test coding. 
(Conta ins one figure, one table, eight appendix tables, and eight 
references.) (SLD) 



* * * >* f * * * * * * * * * * * * * y? * * y? * & i'c s'* * yc * yc v? y? y? vc y? * * y? y? Vc y? y? y? y? y? y? y ? yc y? y? yc yc >v yc yc >v y? y? y? y? y? >v y? y? yc yc >v yc yc 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



CO 



On 

CO 



Q 

w 



u S DEPARTMENT OF EDUCATION 

Office of Educational Research and Improvement 

Z TIONAL RESOURCES INFORMATION 
CENTER (ERICt 

document has been reproduced as 
received from the person or organization 
originating it 

□ Minor changes have been made lo 
improve reproduction quality 

• Points of view ot opinions slated in this 
document do not necessarily represent 
official OERI position or policy 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL 
HAS BEEN GRANTED BY 

%>A/0 T>. Lao a) ft 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



Creating Item Signatures From California’s Mathematics Framework: 
The First Step to Individual Result Reporting 



Presented in New York City at the 
1996 Annual Meeting of the 
American Educational Research Association 
11 April 19% 



by 



John D. Leonard 

University of California at Los Angeles 
Graduate School of Education and Information Studies 



BEST COPY AVAILABLE 



Oo 




ERIC 



My thanks to Jim Mirocha of the UCLA Graduate School of Education who thoughtfully read, made 
suggestions, and provided a much needed sounding board to help me keep the focus on the point of this 
paper. 



u 



Creating Item Signatures From California’s Mathematics Framework: 

The First Step to Individual Result Reporting 

John D. Leonard 1 

University of California at Los Angeles 
Graduate School of Education and Information Studies 

The Background of Changing Assessment 

hematics education is currently undergoing its third major reform in the past 
forty years. This reform, unlike the previous two, has its beginnings with the 
mathematics education community. States, like the mathematics education community, 
have become part of this reform by reasserting their control over the mathematics 
education within their borders through revised or new policy documents patterned after 
the National Council of Teachers of Mathematics (NCTM) Curriculum and Evaluation 
Standards for School Mathematics. As part of this reform, expectations of learning, the 
learning environment, and the tools to assess these facets of this reform are coming 
under scrutiny. California and Vermont, for example, have begun to develop and use 
large scale performance assessments to compare students to standards rather than 
traditional testing which rank orders students. 

In cooperation with the mathematics community, states have included teachers 
and others not usually associated with assessment as part of the process of developing 
different ways of assessing student performance. This inclusion of a diverse population 2 
in test development and the efforts to develop assessments to meet demands quite 
different from standardized, norm reference tests of the past, has created a need for ways 
to differentiate between test forms and characterize test items . Under these new 
conditions of student performance assessment, it becomes critical that students be given 
similar content items with comparable demands of performance in order to compare 
students performance to a standard. In situations where states such as California or 
Vermont have embarked on large scale performance assessments, means to distinguish 
content and performance are essential because the large scale performance assessments 

1- This research was begun under the direction of Leigh Burstein who was the project Director of the CRESST 
School Delivery Standards Project, at the time of his untimely death. It is based on an amalgamation of 
ideas derived from (1) that project’s work with the staff and technical advisory panel from the California 
Learning Assessment System (CLAS), (2) work done on the NSF Validating National Curriculum Indicators 
Project (McDonnell & Burstein; Mirocha, Ormseth, and Guiton as critical contributors ), and (3) the CCSSO 
SCASS Science Assessment Project. 

2 By diverse population, I am referring to parents and teachers. These groups are not usually a part of 
assessment development. 
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often (1) indude multiple tasks and multiple test forms used for each grade and subject 
matter and (2) have a stated obligation to report performance levels for individual 
students when compared to a standard. 

Thus, in the context of performance and content standards assessment 
development assodated with current educational reform, we need a means for 
comparing, first, the content, second, the tasks, and, third, the different forms of a 
performance assessment. Without valid and reliable schemes for accomplishing these 
comparisons, the burden in producing comparable performance levels can lead to over 
reliance on empirical equating methods to compensate for differences in form content 
and difficulty. This over reliance on empirical methods would make it extremely 
unlikely that individual student results would be meaningful. 

The term item characterization is often used to refer to the process of labeling a 
particular test item as to its content or other distinguishing characteristics 3 . In addition, 
the term item signature is used in this paper to refer to the various characteristics to an 
assessment item. When combined, all of the item signatures of a particular form of a 
multiple form assessment, provide a means to make meaningful comparisons between 
multiple forms. By using these signatures as a guide during the constructing of multiple 
forms of an assessments, the need for complex empirical methods should be lessened. It 
is assumed that these methods of characterizing assessment items could also be applied 
to ''Tier forms of assessment such as portfolios or other original documents (artifacts) 
produced by students and used as part of an evaluation scheme. Mathematics 
assessment in California 4 , for example, had several forms of each test at three different 
grade levels with each containing an original artifact produced by students, as well as 
more traditional response items. In this instance, the task of characterizing items by 
content or performance expectation can be viewed as an effort by assessment creators to 
identify the "signature" 5 of a particular item 6 . 

3 When this process is used in reference to artifacts other than test items, for example, student portfolios, 
assignments, or other documents, the term coding is used to indicate the actual labeling. 

4 The CLAS (California Learning Assessment System) was being used when this investigation was 
conducted. 

^The signature concept was developed during the Survey of Mathematics and Science Opportunities Study 
(SMSO, 1993) conducted in connection with the Third International Mathematics and Science Study 
(TIMSS), as a means of applying the TIMSS multi-aspect, multi-dimensional curriculum framework to the 
characterization of the full array of content assessed by individual test items. 

^ Although it is not part of this investigation, there is a need for a language system for describing, or 
preferably, characterizing assessments in a common way applies as well to instructional artifacts 
(classroom assessments, projects, exams, etc.) which have been collected from sites (classrooms, schools) 
attempting to implement reform or to the portfolio systems. There are schemes currently under development 
as an alternative means of monitoring student accomplishment. In these instances, the intent is to code 
artifacts/ portfolios by established or developing rubrics to facilitate characterization of the performance 
they reflect and to provide a means of comparison to other assessment results. 
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Sources For Characterization Schemes 

There are a variety of policy, subject matter, research and standards frameworks, 
documents, and statements that could serve as the basis for characterizing items of 
assessments and / or the coding of instructional artifacts. In academic area of 
mathematics, one could rely on, for instance, any of the following as the starting point 
for characterizing assessment items (creating item signatures). 

1. NCTM Standards (1989); 

2. TIMSS Curriculum Framework (TIMSS)(1993); 

3. The survey categories from the Mathematics teacher survey (‘questions 10, 
11, 12, 15,16,17, and 21 have questions relevant to assessment (content and 
performance expectations) and assessment tasks (types of questions 
emphasized, goals that call for conjecturing, etc. ) of the McDonnell- 
Burstein NSF Validating National Indicators Project; 

4. Reform Up Close Study Dimensions (Wisconsin Center for Education 
Research Porter (Porter, 1993); and 

5. NAEP Achievement Levels Content Descriptors (Sugrue, 1994), 

Daro (Daro, 1993), for example, has argued that any national assessment in 

mathematics must be based on the NCTM Curriculum and Evaluation Standards for 
School Mathematics (1989). Thus, in much the same spirit that Daro has suggested that 
national assessments be judged by the NCTM Standards, it is suggested that individual 
states should be judging their state mathematics program's assessments tools by item 
signatures based on the state's own policy document. Hence, item signatures based on 
the individual state's policy document could produce items closely aligned to the state's 
goals. California, for example, has a mathematics curriculum policy document, 
Mathematics Framework for California Public Schools: Kindergarten Through Grade 
Twelve (CMF), that has served as a driving force to change the assessment system of the 
state's mathematics program and could serve as a basis for item characterizations on 
state's mathematics assessments. 

The Scope of This Study 

This study was conducted to answer three questions: 

(1) Can an item characterization scheme be created based on a state policy document 
when the policy document serves as the driving force behind large scale 
performance assessment? 

(2) Does such an item characterization scheme provide a means to differentiate tasks 
within an assessment and produce equivalent assessment forms at various grade 
levels? 
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(3) Do the signatures based on such an item characterization scheme provide 
information comparable to previously mentioned existing methods of 
characterizing items? 

Satisfying these three criteria, a state assessment scheme could be use to begin to report 
results that are meaningful individual student level without resorting to empirical 
analyses to equate tests containing differing content, performance tasks, etc. 

Rationale 

Because it is the intent of policy documents and state testing to include classroom 
teachers in the process of test development, a characterization scheme based on a state's 
policy document should be more easily used by teachers than the other mentioned 
schemes such as Porter, TEMSS. It is assumed that people working on items for such a 
state testing program would be familiar with the policy document and would therefore 
be able to use such a characterization with minimal training. Hence, an item 
characterization scheme based on the familiar document should shorten the process of 
assessment development. 

The state of California was chosen for this investigation because California has 
such s document in its Mathematics Framework for California Public Schools: 
Kindergarten Through Grade Twelve (1985; 1992) and was beginning to move into the 
arena of large scale performance assessment with its CLAS testing program begun in 
19927. 

Dimensions of Item Characterization 

There is a considerable history of item characterizing of assessments and item 
specifications of test forms in standardized test development. In California, for example, 
the State formed a panel to develop tests for grades 4, 8, and 10. The Performance Level 
Setting Group for the 1993 CLAS Grade 4 Mathematics assessment, drew on that history 
and produced a characterization protocol that included specific task features. The 
following are explanations of the characterization scheme produced from the 1985 CMF. 

a. Strand - Major content strands from math framework. Each assessment 
item may be coded to reflect one of more strands that are represented in the 
item’s content . 

b. Task types - Each item is coded with a CR, SA/OE, or EMC to indicate 
whether it involves the student "choosing a response (CR)", short answer 
(SA), "open-ended" response called for (OE), or bubbling a result into a 
matrix of digits which is considered an "enhanced multiple choice" 
response. 

7 This CLAS program was terminated in 1994 by Governor Pete Wilson after negative public reaction to 
portions of the reading and writing sections of the statewide assessment. 
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c. Performaince expectation/ process characterization — In this portion of an 
item's signature, the task is characterized by the process involved by the 
student in responding to the item. This part of an items signature indicates 
whether the response involves producing writing, "pictures", algorithmic 
procedures, or multiple representations in order to fully respond. 

d. Other Comments — Special features of an item that are worthy of note. 
Using this California example as a specific case of an item characterization 

scheme, it can be generalized and elaborated in order to serve as the basis for identifying 
item signatures as do the previously mentioned schemes (NAEP, TIMSS, or Porter). 
Generalizing, there are four features of assessment tasks that contribute to an item's 
characterization and signature: 

1. Content/ topics — This feature conveys the content dimension of the task expressed 
in conventional terminology. In many of the documents that might be used to 
produce rubrics, the terms "strand" and "unifying idea" better describe the intent of 
this feature. 

2. Item type - MC, open-ended, short answer (CL AS refers to CR/SA, EMC, and 
Open-Ended). 

3. Process /Performance Expectation -- Whether one uses the conventional process 
term or performance expectation doesn’t matter at this point. This feature refers to 
what one does mathematically or otherwise to perform the task rather than content 
of the item. In the reform discussions, this often is characterized as the "-ing" 
activities (conjecturing, comparing, contrasting, ...) 8 . 

4. Other/ miscellaneous - Response mode (picture, graph, written text, ...); the 
linguistic features of the prompt (i.e., judgments about the demands on 
recomprehension and vocabulary separate from specific mathematical content); 
other special features of the tasks that warrant note. 

Developing Signatures, Characterizations and Descriptions 

The signatures or characterizations used in this research examines the results of a 
scheme which was created by a teacher /researcher. The scheme devised was based on 
the previous work of the 1993 CLAS Performance Level Setting Group and the CMF. 
Once created, a panel of four individuals familiar with mathematics and the CMF were 
trained in one thirty minute session on the use of the scheme to create an individual 
item's signature. In subsequent meetings, individual item signatures for actual items 



8 One classic split is conceptual/ declarative knowledge, procedural knowledge, and problem solving . 
Another alternative is to think of this dimension as a means of describing the cognitive demands of the task 
itself (what one has to know to be able to sensibly attempt the task) or the response the task engenders (in 
other words, what kind of response is called for). 
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from the 1993 CL AS examinations were made by the each member of the panel working 
independently. The resulting descriptions were compared and modifications to the 
scheme were made based on the suggestions of the panel.. 

Based on the discussions of this panel and the resulting item signatures of the 
characterization scheme, it was concluded that the CMF could be used as a basis for 
creating item signatures. A complete description of the signature criteria follows. 

Item Characterization Scheme Based on the CMF 

The final item characterization used in this investigation uses six components in 
an item’s signature. These characterizations when combined form the signature of an 
individual item. 9 The categories or components used are: 

a. Strand: The CMF mentions eight 10 content categories the permeate the 
mathematics program and have appeared in California mathematics 
frameworks for about 30 years. Each item may fit into one or more of these 
strand content categories: Functions, Algebra, Geometry, Statistics and 
Probability, Discrete Mathematics, Measurement, Number, and Logic and 
Language. A detailed description of these strands can be found on pages 75-87 
of the CMF (1992). 

b. Unifying Ideas: A unifying idea is a major mathematical theme relevant to or 
spanning several different strands. These unifying ideas emerge when a 
broader view of content is taken. Unifying ideas also indicate the depth of 
understanding appropriate to a grade level. There are ten unifying ideas with 
only a few at each of the grade-level groupings (K-5, 6-8, and 9-12). While each 
of the unifying ideas is relevant at later stages, the most emphasis and 
prominence are given to the unifying ideas being newly introduced at the 
grade level grouping (1992). These unifying ideas are discussed at each grade 
level in th? CMF (grades K-5: pages 108-109; grades 6-8: pages 122-125; and 
grades 9-12: pages 158-163). 

c. Task Type: The task type refers to the type of response solicited from the 
student. The categories used for the CLAS tests are OE for open-ended 
response item, MC for multiple-choice response item, EMC for extended 
multiple-choice item, and OTL was used to indicate an item assessing student 
reported "opportunity to learn" information. The EMC items require a 
student to ,r bubble" in the correct numerical answer on a grid of numbers 
rather than choose a given complete numerical answer. 

9 These are based on tho 1985 and 199. CMF's description of mathematics in grades K-12 envisioned in these 
documents. 

1 0 The eight strands are described on pages 80-87 of the 1992 CMF. 
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d. Performance Expectation: The CMF uses the phrase mathematical power 
which is described in this way: Mathematically powerful students think and 
communicate , drawing on mathematical ideas and using mathematical tools 
and techniques. Based on the further description in the CMF, these 
performance expectations can be used to characterize an assessment item 11 : 

• Thinking refers to intellectual activity and includes analyzing, classifying, 
planning, comparing, investigating, designing, inferring and deducing, 
making hypotheses and mathematical models, and testing and verifying 
them. This expectation is not coded if an item involves applying an 
algorithm to a given situation where the activity consists of recognizing a 
situation for the use of the algorithm. 

• Communication refers to coherent expression of one's mathematical 
processes and results. This is not coded in the case where a student is 
required to mark an answer. 

• Ideas refers to content: mathematical concepts such as addition, 
proportional relationships, geometry, counting, and limits. 

• Tools and techniques extend from literal tools such as calculators and 
compasses and their effective use to figurative tools such as 
computational algorithms and making visual representations of data. 

e. Linguistic complexity: While the CMF makes no mention of linguistics, the 
concern for equity is prominent in the framework as noted concisely in the 
phrase, "... this document reasserts the goal of mathematical power for all 
students and emphasizes the phrase 'for all students.' ... giving every student 
in California fair access to mathematics education. Included are females and 
males; rich, poor, and middle class; descendants from all parts of the world; 
speakers of Mandarin, English, Arabic, Spanish, and the more than 200 other 
first languages of U. S. citizens." As a consequence of this, acknowledgment of 
the diversity of the California student population, it was deemed necessary to 
make some note of the linguistic feature of an item. A simple way is to 
characterize the linguistic complexity of an item is with an index of 'easily 
accessible for the grade level (EA.)', 'accessible for majority of grade level (A)'; 
or 'not accessible for most of grade level (NA)' for which the test is 
administered. This index and the term 'accessible' is one that is generally 



1 1 There are problems with this category which will be discussed at the end of this paper. 
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accepted by members of the Language Arts community and was used in the 
absence of others being currently developed. 12 
f. Other Comments: On occasion, an item is found to include phrases or words 
that might have either a cultural bias or might not be generally understood by 
the student population. This characterization category was included to alert 
those constructing items of possible problems. During sample coding sessions, 
remarks were included to refer to content when an item did not appear to fit 
into one or more of the previous signature categories. 

For example, an item similar to the one listed in figure 1 was assigned this 
signature 13 : 

(a) STRAND: Statistics and probability 

(b) UNIFYING IDEAS: Multiple representation & Finding, making, 

and describing patterns 

(0 OE: Open ended task 

(d) PERFORMANCE EXPECTATION: Thinking; Communication; Tools and 

techniques 

(e) LINGUISTIC COMPLEXITY: A 

(f) OTHER COMMENTS: None 

Figure 1 - (Similar to Eighth Grade CLAS Item) 

Information is collected and used for many different reasons. Some examples are: 

• A company plans to report if there has oeen an increase in the recycling of 
newspapers. 

• A company that makes bicycles wants to predict sales. 

• The school PTA operates a stwck bar at lunch and after school and wants to know 
how tnany of each kind of snacks to order. 

You are responsible for uniting a report to your class on ONE of the examples listed above. 

Explain how you would GATHER and ORGANIZE the information, and report your 
findings . You may use graphs, tables , or words to explain your answer. 

Results of CMF Coding of Items and Forms 

After the initial revision, the characterization scheme produced the same 
signatures on 85% of the items of the assessment (80 items were examined on the eight 
forms of one level of the test). With the 15% that were not identical, four out of the five 
panelists agreed with the exception of one item which had only three of the five in 



12 Jamal Abedi is examining the linguistic features of mathematics items on assessments. This work may 
provide more descriptive ways to characterize items and, ultimately, provide the tools to create items that 
are easily accessible to the majority of students. 

13 The item was originally paraphrased to maintain the confidentiality of the examination. 
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agreement. With discussion, agreement was reached on all the remaining disputed 
items. These results satisfied the first question of this study: an item characterization can 
be created from a state policy document. In this study, the policy document used for the 
source of the item characterization scheme was the California Mathematics Framework. 

While a single item's signature does not reveal the entire scope of an assessment, 
creating a matrix of ?ach of the multiple forms of an assessment such as the Grade 8 
CLAS test, does provide an indication of what student performance has been assessed by 
the sum of the items of that particular form of the assessment. Furthermore, with 
matrices of the multiple forms at one level, test designers can make between form 
comparisons and decisions about -the comparability of *he forms based on these item 
signatures. In the case of the Middle Grades Performance Assessment in Mathematics 
(Grade 8 CLAS test), comparison of the eight forms reveals that the multiple forms 
varied considerably on all categories thus making the task of having meaningful 
individual result reporting unlikely. 

While coding individual items on samples of the assessment, coders were not 
aware of these differences. Once the matrix of the ten items of a form was created, over- 
and under- representations in each signature category became apparent. When all the 
form matrices were examined, it became apparent that (1) items needed to be moved 
between forms in order to balance various categories and (2) some items would need to 
be reworded to reflect the appropriate grade level emphasis stated in the CMF. Thus, had 
this signature scheme been in place, many of the criticisms directed at the test and, 
hence, indirectly at the committee that assembled the test, could have been avoided. It 
should be noted that in defense of the groups that created and assembled the various 
forms at the three levels, this was a concern that was expressed. But, unfortunately, in 
this instance, it appears that the committee had neither the time nor resources to deal 
with it. As with any new assessment, there is a learning curve associated with 
complexity of creating comparable test forms. 14 This last finding served to answer the 
second question of the study: such an item characterization does provide a means to 
compare tasks and could be used to produce equivalent forms of a large scale 
performance assessment. 

A sample of the panel’s coding of one grade level of the test is included in 
Appendix A. Also included in Appendix A is the same form coded with three other 



The original expectation of the CLAS test was to produce first a performance based assessment that 
would monitor the successful adaptation of the mathematics program to the goals of the CMF. Second, the 
goal was to produce a test with individual student reportability. This process was in place as wa 
proceeding toward the goal of individual student reporting when the program was canceled. 
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characterization schemes to indicate the utility of each. Preceding each coding is an 
explanation of the characterization scheme. 

A Comparison of the CMF Signatures to Existing Methods of Item Characterization 

As mentioned previously, the final purpose of this investigation was to compare 
this signature scheme to existing item characterization schemes. For the purpose of this 
investigation, three other methods were used in this comparison. One was based on 
work by McDonnell & Burstein, a second using Porter's content dimensions, and the 
third using TIMSS signatures. Each of these coding schemes is described more fully in 
Appendix A. To facilitate comparisons between coding schemes and illustrate each, the 
item signatures for the question described in Figure 1, are shown in Table 1. 

Observing the item's signature in each of the categories, a through f, gives an 
indication of the similarities of the various characterization schemes. It can be seen that 
each scheme gives an indication of the concent of an item, the item response required, a 
description of the student process or performance, and includes comments as 
appropriate. Three signatures (McDonnell-Burstein, TIMSS, and Porter) have a much 
more comprehensive coverage of content as opposed to the scheme developed for this 
study. However, the CMF coding scheme, does have a category corresponding to the 
Unifying Ideas section of the framework. All four schemes reveal information about 
the particular item and, when combined with the signatures for all of the items of a 
particular form, would allow judgments to be made about the comparability among 
multiple forms in the categories of content and performance expectation. Since it is the 
purpose of a large scale performance assessment to produce valid reporting that avoids 
over reliance on empirical equating methods to compensate for differences in form 
content and performance expectations, it would be a first step to make use of an item's 
signature to build a particular form's signature that would produce forms with smaller 
differences in content and performance expectations. 

To further illustrate the differences between characterization schemes and 
provide examples of the information to be gained from such form characterizations. 
Appendix B contains the coding for three forms of the Grade 8 ClJ S Mathematics 
Assessment using the four item characterization schemes mentioned. When these 
three form’s item signatures examined, all four schemes do show that there is 
variation between the three assessment forms in terms of content and performance 
tasks which was the third part of this research. This variation would increase the 
difficulty of meaningful individual reports. Thus, the characterization scheme based on 
the CMF does provide (1) a means for assessment panels to make grade level forms 
more representative of the goals of a particular grade level assessment and (2) a means 
to make forms within a grade level more equal in content and task demands. 
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Table 1 



CMF: 


TIMSS: 


McDonnell- 

Burstein: 


Porter: 


(a) Statistics and 


(a) Data representation 


(a) Statistics and 


(a) Number and 


probability 


(1.7.1) ; Validation 

(1.9.2) 


probability 


number relations 
(0.9), Statistics(6.0) 


(b) Multiple 


(b) Not applicable to 


(b) Not applicable in 


(b) Not applicable to 


representation & 
Finding, making, 
and describing 
patterns; 


the TIMSS coding 


McDonnell- 
Burstein coding. 


Porter's coding 


(c)OE; 


(c)OE; 


(c) OE; 


(c) OE; 


(d) Thinking; 


(d) Knowing (2.1.1), 


(d) Explain(d), 


(d) Interpret data (6) 


Communication; 


PrblmSlvng (2.3.2, 


Writing equations 




Tools and 
techniques; 


2.3.2), 

Communication 

(2.5.3) 


(sss) 




(e) A 


(e) Not used. 


(e) Not used. 


(e) Not used. 


(f) No Comments. 


(f) Students are to 
make a choice of 
which to answer. 


(f) No Comments. 


(f) Students are to 
make a choice of 
which question to 
answer. 



Conclusions 



While each of the coding schemes provides differing information about 
individual items and forms through their signatures, each provides information that is 
valuable to test makers attempting to create performance assessments with comparable 
content and tasks. No one scheme has a clear advantage over another since each was 
designed for use by a specific group. In the case of a state, such as the case of California, 
there is a twofold advantage to using the policy document as the basis for item 
characterization. First of all, there would be the familiarity with the CMF which would 
theoretically shorten the time needed to create item signatures. Secondly, by lessening 
the development time, a cost savings to the assessment program should be realized. 
Finally, since such a coding scheme would be an extension of the ideas contained in the 
CMF, it would be more understandable to the state's mathematics education 
community and provide a model for individual districts in there own testing 

13 
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development 15 . Once this scheme is more widely disseminated within a state, 
individual student scores can be reported and more easily understood. 

Implication for Future Work on Policy Document Based Item Signatures 

Based on the results of this investigation, it may be productive for individuals or 
groups of individuals in other states to develop signature schemes based on their state's 
policy documents if such a document exists. It might also be productive to create a item 
characterization scheme based n the NCTM Standards that could serve as a basis for 
individual state schemes. 

In terms of the signature scheme developed in this investigation, there are 
ambiguities within the "unifying ideas", "performance expectation", and "linguistic 
complexity" categories. These ambiguities could be resolved by creating or further 
defining subcategories within each. The TIMSS characterization scheme may provide a 
useful starting place for deliniating content or the unifying idea characterization if this 
is desired. Linguistic complexity could be more clearly defined using schemes 
mentioned in the footnotes (Abedi, 1994). 

All of the panelists agreed at the end of the sessions used to code the various 
forms that the performance expectation category "thinking" proved to be the least 
useful in the coding process. The description of the "thinking" category given in section 
d, could be used to further define the performance that a student's response should 
exemplify. 



15 Individual school districts in California are required to have a profit.ency test which students must pass 
as a graduation requirement. 
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3 Original item, 16 a., reads "Items that require students to recognize or recall definitions of concepts" 

4 Original item, 23 q., reads "Writing about mathematical ideas" 
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5 The column headings, (a),...,(f), are those listed in "Characterizing Mathematics Assessments, Artifacts, and Portfolios (Signatures)" a work in progress from 
CRESST at UCLA. 

** The Linguistic complexify is denoted on a three point scale for the grade level: "Easily accessible, Accessible, Limited accessibility". 
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7 The column headings, (a ),..., (f), are those listed in "Characterizing Mathematics Assessments, Artifacts, and Portfolios (Signatures)" a work in progress from 
CRESST at UCLA. 

8 The Linguistic complexity is denoted on a three point scale for the grade level: "Easily accessible. Accessible, Limited accessibility". 
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from CRESST at UCLA. 

U The Linguistic complexity is denoted on a three point scale for the grade level: "Easily accessible, Accessible, Limited accessibility". 
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